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Attorney Docket No: 21295.59 (H5644US) 

METHOD, ARRANGEMENT, AND SOFTWARE FOR OPTIMIZING THE 
IMAGE QUALITY OF MOVABLE SUBJECTS IMAGED WITH A 

MICROSCOPE 

CROSS REFERENCE TO RELATED APPLICATIONS 

5 This application claims priority of the German patent application 

102 35 657.2 which is incorporated by reference herein. 

FIELD OF THE INVENTION 

The invention concerns a method for optimizing the image quality 
of image sequences of movable subjects acquired with a microscope. 

10 The invention further concerns an arrangement for optimizing the image quality of 
image sequences of movable subjects acquired with a microscope. 

BACKGROUND OF THE INVENTION 

In the observation of living and movable subjects, artifacts occur in 
image production because the subjects move. This on the one hand results in 
15 unsharpness (motion generates artifacts similar to those of defocusing), and on the 
other hand, in confocal microscopy the images exhibit poor quality (signal-to- 
noise ratio) because methods such as image averaging cannot be applied to a pixel 
when motion is present. With averaging, for example, motion would cause subject 
pixels to be mixed with non-subject pixels. 

20 SUMMARY OF THE INVENTION 

It is the object of the invention to create a method with which it is 
possible to generate high-quality images of movable subjects, and to make 
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possible efficient application of operations such as averaging and filtering, even to 
movable subjects. 

The object is achieved by means of a method for optimizing the image quality of 
movable subjects imaged with a microscope system, comprising the following 
steps: 

a) acquiring a plurality of images having a plurality of pixels; 

b) determining a respective displacement vector field from a comparison 
of the pixels of each two chronologically successive images; 

c) identifying a trajectory for each pixel of the image from the 
displacement vector fields; and 

d) applying an operation to the image data along a trajectory. 

A further object of the invention is to create an arrangement with which it is 
possible to generate high-quality images of movable subjects, and to make 
possible efficient application of operations such as averaging and filtering, even to 
movable subjects. 

The object is achieved by way of an arrangement for optimizing the image quality 
of movable subjects imaged with a microscope system, the microscope system 
comprising: at least one objective defining an image window, a detector unit for 
acquiring a plurality of images each having a plurality of pixels, a computer 
system, which encompasses a means for determining a respective displacement 
vector field from a comparison of the respective pixels of at least two 
chronologically successive images, a means for identifying a trajectory for each 
pixel of the image from the displacement vector fields, and a means for applying 
an operation to the image data along a trajectory. 

In order to solve the problem associated with these objects, it is advantageous that 
a trajectory, which records displacements and thus subject motions, is determined 
for each pixel of the image. The displacements and subject motions are 
advantageously determined as displacement vector fields which evaluate in their 
totality all of the motions within the scene. The displacement vector field results 
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from a comparison of the pixels of, in each case, at least two chronologically 
successive images. The use of more than two images of a sequence may result in 
better convergence. Such displacement fields are determined by solving a flow 
problem, a pixel change model being formulated as a differential equation and 
5 fitted numerically to the image data using a minimum description length (MDL) 
method. Probably the most prominent representative of such models is the 
modeling of the motion of solid bodies in video technology, for which the 
synonym "optical flow method" has already become established. Further 
representatives may be found, for example, in climate modeling, where liquid 

10 bodies (from clouds to water) are modeled. Although the "optical flow" 
designation is not common here, this text uses the term synonymously. A 
trajectory is constructed by tracking the local displacement vectors from pixel to 
pixel, which can easily be accomplished with a computer algorithm. The 
trajectory determined in this fashion is a so-called guideline for the application of 

1 5 operations. Operations along the trajectory that is determined can be, for example 
(with no limitation as to generality), a deconvolution, a smoothing, or an 
averaging filter. An extension to the entire class of image processing classes 
operating in time-lateral fashion is included in this application in this context, and 
is left to the imagination of one skilled in the art in terms of implementing a 

20 system. 

A peculiarity of these new operations is that ambiguities occur as a result of the 
motion and the displacement vector field. For example, a subject can migrate into 
the image segment, and a filter with memory must treat that new pixel differently 
from another, more-static pixel in the same scene. Another example is the splitting 
25 of a subject into several subjects (trajectory source). Yet another is the 

combination of individual pixels into one (trajectory sink). This is solved by way 
of intelligent trajectory management. 

Further advantageous embodiments of the invention are evident from the 
dependent claims. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The subject matter of the invention is depicted schematically in the 
drawings and will be explained below with reference to the Figures, in which: 

FIG. 1 schematically depicts a scanning microscope; 

FIG. 2 schematically depicts the image frame imaged through the microscope, 
and the manner in which it is subdivided into individual regions or 
pixels; 

FIG. 3 schematically depicts the processing of the data obtained from the 
observation of living and movable subjects; 

FIG. 4 is a block diagram of the method according to the present invention; 

FIG. 5 depicts an example of a situation in which a subject leaves the image, 
and the identified trajectory ends at the edge; 

FIG. 6 depicts an example of a situation in which a subject comes into the 
image; 

FIG. 7 depicts a situation in which a subject splits and several trajectories 
result therefrom; and 

FIG. 8 depicts a situation in which several subjects combine into one subject, 
and the trajectories of the individual subjects end at one point. 

DETAILED DESCRIPTION OF THE INVENTION 

FIG. 1 schematically shows an exemplary embodiment of a 
confocal scanning microscope system with which the method according to the 
present invention can be carried out. Other microscope systems can likewise be 
used. A detector unit 19 is equipped with a video system or a CCD sensor for 
acquiring images. 
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This is not, however to be construed as a limitation of the invention. It is 
sufficiently clear to one skilled in the art that the invention can also be carried out 
with conventional microscopes with digital image production. Illuminating light 
beam 3 coming from at least one illumination system 1 is directed by a beam 
splitter or a suitable deflection means 5 to a scanning module 7. Before 
illuminating light beam 3 strikes deflection means 5 5 it passes through an 
illumination pinhole 6. Scanning module 7 comprises a gimbal-mounted scanning 
mirror 9 that guides illuminating light beam 3 through a scanning optical system 
12 and a microscope objective 13, over or through a subject 15. In the case of non- 
transparent subjects 15, illuminating light beam 3 is guided over the subject 
surface. With biological subjects 15 (preparations) or transparent subjects, 
illuminating light beam 3 can also be guided through subject 15. For that purpose, 
non-luminous preparations are optionally prepared with a suitable dye (not 
depicted, since established existing art). The dyes present in the subject are 
excited by illuminating light beam 3 and emit light in a characteristic spectral 
region peculiar to them. This light proceeding from subject 15 defines a detected 
light beam 17. The latter travels through microscope optical system 13 and 
scanning optical system 12 and via scanning module 7 to deflection means 5, 
passes through the latter and arrives, through a detection pinhole 18, at at least one 
detector unit 19, which is equipped in the exemplary embodiment depicted here 
with at least one photomultiplier as detector. It is clear to one skilled in the art that 
other detectors, for example diodes, diode arrays, photomultiplier arrays, CCD 
chips, or CMOS image sensors, can also be used. Detected light beam 17 
proceeding from or defined by subject 15 is depicted in FIG. 1 as a dashed line. In 
detector 19, electrical detected signals proportional to the power level of the light 
proceeding from subject 15 are generated. Since, as already mentioned above, 
light of more than one wavelength is emitted from subject 15, it is useful to insert 
in front of detector unit 19 a selection means 21 for the spectrum proceeding from 
the sample. The data generated by detector unit 19 are forwarded to a computer 
system 23. At least one peripheral unit 27 is associated with computer system 23. 
The peripheral unit can be, for example, a display on which the user receives 
instructions for adjusting the scanning microscope and can also view the present 
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setup and also the image data in graphical form. Also associated with computer 
system 23 is an input means comprising, for example, a keyboard 28, an adjusting 
apparatus 29 for the components of the microscope system, and a mouse 30. 

FIG. 2 schematically depicts an image frame 41 acquired with microscope 100. 
5 Image frame 41 is defined by the image window determined by microscope 100. 
Image frame 41 is subdivided into individual regions or pixels 39. Movable 
subject 40 is located within image frame 41. Pixels 39 can be embodied as two- 
dimensional regions of image frame 41, or also as three-dimensional regions of 
image frame 41. 

10 FIG. 3 shows the observation of living and movable subjects 40 and the 

processing of data obtained from the observation of living and movable subjects 
40. For the observation of living and movable subjects 40, several images or 
image frames 41 1, 41 2 , 4b, 41 n are acquired consecutively, for example using 
scanning microscope 100 described in FIG. 1, each image frame 41 1, 41 2 , 413, 

15 41 n defining an XY plane or an acquired specimen volume XYZ. Between each 
two successive images, e.g. 41 1, 41 2 or 41 2 , 41 3 , or 41 n -i» 41 m a respective 
displacement vector field 42\ 9 42 2 , 42 n -i is determined. The displacement 
vector field between two successive images, e.g. 41 2 and 4I3, can be determined 
from a comparison of the individual mutually corresponding pixels of the two 

20 images. Proceeding from a first image 41 1 having N pixels, it is thus possible to 
ascertain the new positions in the next image 42 2 by way of the displacement. An 
even more accurate model can also be fitted for a trajectory 43, with sub-pixel 
accuracy, from the discrete displacements. Advantageously, more than one 
successive image is then used for this accuracy-enhancing operation. Trajectory 

25 43 for the movable subject is obtained from the plurality of displacement vector 
fields 42i, 42 2 , 42 n -i by tracking the displacement vector fields of the 
individual images 41i, 41 2 , 41 3 , 41 n . In the graphical depiction of trajectory 43, 
the moving subjects are represented by at least one trajectory through XYt space 
44. 
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A video contains a three-dimensional space-time (two spatial dimensions XY, one 
time dimension t). The pixels of a movable subject 40 thus move along a curved 
path (trajectory) within this space-time. Trajectory 43 that is determined defines 
this curved path unequivocally, and data concerning the motion of subject 40 are 
thereby obtained. Operations that are to be applied to the moving subject can thus 
be performed along trajectory 43. For example, data about said trajectory 43 can 
be fed to an averaging filter, yielding an image of higher quality that takes into 
account the motion of subject 40, specifically in that the signal-to-noise ratio is 
better. This approach is of course also possible for sequences of volumes (four- 
dimensional space-time), and can be transferred to any kind of operation, e.g. 
filters (deconvolution, smoothing). In order to produce these filters, instead of the 
simple summing formulas common in image processing, the continuous operation 
equation must be discretized to the trajectory in the curved space-time, 
incorporating the present geometry. Such methods are established in numerical 
mathematics, and are existing art in simulation technology. 

FIG. 4 is a block diagram of the method according to the present invention. The 
first step is image acquisition 50 of a series of images. As already described 
above, acquisition is accomplished using detector unit 19 of the microscope or 
scanning microscope. The data representing each image are stored in a first image 
memory 52. From image memory 52, the images are sequentially conveyed to an 
optical flow calculator 53. Parallel therewith, the data of each image are conveyed 
to a nonlinear filter 54. From optical flow calculator 53, the data modified by 
calculator 53 are conveyed to a trajectory tracker 55 and then to a trajectory 
memory 56. The data present in trajectory memory 56 are also made available to 
nonlinear filter 54 in order to allow discretization. As already mentioned above, 
any desired operations are applied to the acquired image data, taking place in 
nonlinear filter 54 in consideration of the stored trajectory 43. The data modified 
in this fashion travel into a second image memory 58 and can be retrieved from 
there, for example, for presentation on a display. 

FIGS. 5 through 8 depict various events that result in respectively distinguished 
trajectories. FIG. 5 depicts the situation in which subject 40 leaves image frame 
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41 during the recording of N image frames. Upon recording of the (N+l)th image 
frame, the subject can no longer be captured by the microscope. Trajectory 43 
resulting from the N captured image frames ends at the edge of XYt space 44. It 
can be deleted from trajectory memory 56 by trajectory tracker 55. 

5 FIG. 6 depicts the situation in which a subject 40 is present in first image frame 
41 1. During the recording of N image frames, a further subject 60 enters the region 
of the image frame, so that it can be captured by the microscope. Subject 60 can 
also be captured by the microscope when the Nth image frame is recorded. In 
addition to trajectory 43 for subject 40, a further trajectory 63 is added in XYt 
1 0 space 44 for subject 60 that has just entered the image frame of the microscope. 

FIG. 7 depicts the situation in which a subject 70 is present in first image frame 
41 1. By the time the Nth image frame is recorded, subject 70 has split into, for 
example, four subjects 70 1, 70 2 , 70 3 , and 70 4 . Subjects 70 \, 70 2 , 70 3 , and 70 4 can 
also be captured by the microscope upon recording of the Nth image frame. In 
15 addition to trajectory 43 for subject 70, four further trajectories 73i, 732, 73 3 , and 
73 4 are added at an end point of trajectory 43 at a certain time t, representing the 
motions of four subjects 70j, 702, 70 3 , and 70 4 . 

FIG. 8 depicts the situation in which four subjects 80j, 80 2 , 80 3 , and 80 4 are 
present in first image frame 41 1. Upon recording of the Nth image frame, subjects 
20 80 1, 8O2, 80 3 , and 80 4 have combined into one subject 80. Trajectories 83 1, 83 2 , 
83 3 , and 83 4 end in XYt space 44 at a point 84. 

The invention has been described with reference to a particular exemplary 
embodiment. It is self-evident, however, that changes and modifications can be 
made without thereby leaving the range of protection of the claims below. 

25 
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